TensorFlow Learning Note

Posted on | In ,

Initialization

shape = [batch,height,width,channel]
constant:

1
2
3
4
tf.zeros(shape)
tf.constant(0.4, shape)
tf.random_norm(shape, stddev=1e-1)
tf.truncated_norm(shape, stddev=1e-1)

variable:

1
2
3
4
5
tf.get_variable(varname, shape=[5,5,64,192], initializer=XXX)
initializer=tf.zeros_initializer() # bias
initializer=tf.random_normal_initializer(0.0, 0.02) 
initializer=tf.truncated_normal_initializer(0.0, 0.02)
initializer=tf.contrib.layers.xavier_initializer() # weight

Empirically, for weight initialization, xavier is better than truncated_norm, better than random_normal.

Summary for tensorboard

1
2
3
4
tf.summary.scalar('total_loss', total_loss) #scalar, histogram, etc
self.merged = tf.summary.merge_all()
train_writer = tf.summary.FileWriter(summary_folder, model.sess.graph) #include the sess graph
train_writer.add_summary(summary, ibatch)

after running the code, open tensorboard tensorboard --logdir='./train'

Checkpoint

1
2
3
4
5
6
7
8
9
10
# save all the variables
saver = tf.train.Saver() 
# save partial variables 
saver = tf.train.Saver({"my_v2": v2})
# save and restore
saver.save(sess, "/tmp/model.ckpt")
saver.restore(sess, "/tmp/model.ckpt")
# restore partial variables
init_fn = slim.assign_from_checkpoint_fn(checkpoint_path, slim.get_variables_to_restore())
init_fn(sess)

inspect the checkpoint file

1
2
3
4
5
6
7
from tensorflow.python import pywrap_tensorflow  
checkpoint_path = os.path.join(model_dir, "model.ckpt")  
reader = pywrap_tensorflow.NewCheckpointReader(checkpoint_path)  
var_to_shape_map = reader.get_variable_to_shape_map()  
for key in var_to_shape_map:  
    print("tensor_name: ", key)  
    print(reader.get_tensor(key)) # Remove this is you want to print only variable names

Layer

  • convolutional layer

    1
    2
    3
    4
    kernel = tf.Variable(tf.zeros([5,5,64,192]))
    tf.nn.conv2d(x, kernel, strides=[1, 1, 1, 1], padding='SAME') # padding is 'SAME' or 'VALID' depending on stride is 1 or larger

    tf.nn.bias_add(prev_layer, bias)

  • dropout layer

    1
    2
    3
    keep_prob = tf.placeholder(tf.float32)
    # set keep_prob as 0.5 during training and 1 during testing
    tf.nn.dropout(h_fc1, keep_prob)

  • pooling layer

    1
    tf.nn.max_pool(x, ksize=[1, 2, 2, 1],  strides=[1, 2, 2, 1], padding='VALID')

  • loss layer

    1
    2
    tf.nn.softmax_cross_entropy_with_logits(labels=y_, logits=y))
    	tf.nn.l2_loss(prev_layer)

  • metric layer

    1
    2
    # calculate top-k accuracy
    tf.nn.in_top_k(logits, label_holder, k) # the accuracy of top k
  • activation layer
    1
    tf.nn.relu(prev_layer)
  • other layers
    1
    2
    3
    tf.one_hot(self.input_class, self.n_class)
    tf.nn.embedding_lookup(codebook, indices)
    tf.nn.lrn() # legacy

Common Operations

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
tf.matmul
tf.pow
tf.diag_part
tf.reduce_sum
tf.reduce_mean
tf.nn.softmax
tf.tile(a,(2,2)) 
tf.nn.embedding_lookup
sim.flatten
tf.squeeze
tf.reshape
tf.concat(values=[a,b,c], axis=3)
tf.gather  #get indexed value, only on 0-dim
tf.gather_nd
tf.map_fn
tf.scan_fn
tf.fold1/foldr

Training and testing

  • training stage

    1
    2
    3
    4
    optimizer = tf.train.GradientDescentOptimizer(0.01)
    train = optimizer.minimize(loss)
    for i in range(1000): #note the iteration number here
        sess.run(train, feed_dict={x:x_train, y:y_train})

  • testing stage

    1
    2
    acc.eval(feed_dict={x:x_test, y:y_test})
    sess.run(acc, feed_dict={x:x_test, y:y_test}

Queue

  • For plain TensorFlow, reading data via queue is cumbersome. Refer to my another blog ‘TensorFlow Input Data’ for more details.
    ``python
    with tf.Session(config=config) as sess: 

      coord = tf.train.Coordinator()
  threads = tf.train.start_queue_runners(sess=sess, coord=coord)
  sess.run(tf.initialize_local_variables())
  #main code
  coord.request_stop()
  coord.join(threads)

    1
    2
    3
    4
    5
    6
    7
    8
    9
    * With tfrecord and slim, using queue is much easier by using 	`slim.queues.QueueRunners`
    	```python
    	with tf.Session(config=config) as sess:        
    		with slim.queues.QueueRunners(sess):
    	``` 
      or `sv.managed_session`
    	```python
    	sv = tf.train.Supervisor(logdir=logdir, save_summaries_secs=0, saver=None)        
    	    with sv.managed_session() as sess:

    Sample code of generating tfrecord dataset and reading data via queue can downloaded here

Utils

  • check tensorflow version

    1
    tf.__version__.split('.')[0] != "1"

  • count parameter numbers

    1
    parameter_count = tf.reduce_sum([tf.reduce_prod(tf.shape(v)) for v in tf.trainable_variables()])